In a fast paced development environment when dealing with a highly complex cloud based service solution, high software quality is quintessential. For any service based upon a cloud infrastructure, the biggest challenge is delivering high quality software updates and enhancements in short release cycles. The complex architecture of a cloud based service solution, large number of features, and a fast paced release cycle make testing a very challenging task. What adds to these challenges is the fact that any software glitch that impacts the cloud infrastructure, effects the full set of users, unlike conventional software solutions. The roles and responsibilities of Quality Assurance (QA) are amplified in such an environment with a zero margin for error. Also, with shorter release cycles, QA has less time to verify the quality of the release.
A cloud based service solution, like Zscaler (www.zscaler.com), is a complex composition of a number of services and features. Huge amounts of effort are needed to perform software testing and achieve an acceptable software quality. A fast paced development cycle and ever increasing number of features being added to the base product with every release makes this effort tougher with time. At one point in time, regression of old features itself becomes a mammoth task, and QA has to spend most of its time performing regression. Apart from this, every change being made to the software introduces the risk of side effects and the software has to go through quality tests many times during a release cycle. This approach would not scale unless complimented with automation testing to take care of regression and sanity tests. In order to achieve a high quality software release in the cloud infrastructure, it is not enough to rely solely upon manual testing. Automation takes a high priority with the growing number of features, considering the complexity of the cloud and shorter release cycles. Automation is a good servant but a bad master. If Automation is not managed properly, it will result into more efforts and no results.
There are three main challenges in automation, namely 1) maintenance of automation scripts and test infrastructure, 2) ease of automation usage, and 3) interpretation of test results. For maintenance of automation scripts and test infrastructure, in the real world, it is never automate and forget. Test Scripts always need to be tested and maintained to work against latest changes made in the cloud. There is a requirement for lots of planning and effort with good future insight. For ease of automation usage, automation infrastructure for a software product, such as for a highly complex cloud based service solution, is not simple in nature. Most of the time, it needs a high level of acquaintance with the scripts and a minimal level of programming knowledge in order to run and interpret automation test results. Hence, only a limited number of people are enabled to use automation effectively, removing the advantages automation is supposed to provide. For interpretation of test results, in order to effectively assess the quality of software, it is very important to effectively interpret the test results. This involves a comparison of results between different components across different software versions, understanding the impact of failures on the overall quality and usage of the software, and scoping the amount of effort needed to fix these issues. This becomes a challenge for a complex software product where a number of factors, both internal and external, influence the results.